<!DOCTYPE html>
<html xmlns:MadCap="http://www.madcapsoftware.com/Schemas/MadCap.xsd" class="_Skins_HTML5___Top_Navigation" lang="en-us" xml:lang="en-us" data-mc-search-type="Stem" data-mc-help-system-file-name="Default.xml" data-mc-path-to-help-system="../../" data-mc-has-content-body="True" data-mc-target-type="WebHelp2" data-mc-runtime-file-type="Topic;Default" data-mc-preload-images="false" data-mc-in-preview-mode="false" data-mc-toc-path="">
    <head>
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <meta charset="utf-8" />
        <meta http-equiv="X-UA-Compatible" content="IE=edge" />
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <link href="../../Skins/Default/Stylesheets/Slideshow.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/TextEffects.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Topic.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Components/Styles.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Components/Tablet.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Default/Stylesheets/Components/Mobile.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/foundation.6.2.3.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/Styles.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/Tablet.css" rel="stylesheet" type="text/css" data-mc-generated="True" />
        <link href="../../Skins/Fluid/Stylesheets/Mobile.css" rel="stylesheet" type="text/css" data-mc-generated="True" /><title>Getting Started</title>
        <link href="../Resources/Stylesheets/Styles.css" rel="stylesheet" type="text/css" />
        <script src="../../Resources/Scripts/custom.modernizr.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/jquery.min.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/require.min.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/require.config.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/foundation.6.2.3_custom.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/plugins.min.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapGlobal.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapDom.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapUtilities.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapXhr.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapTextEffects.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapSlideshow.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapFeedback.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapDefault.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapHelpSystem.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.Breadcrumbs.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.MiniToc.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapToc.SideMenu.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapIndex.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapGlossary.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapParser.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapSearch.js" type="text/javascript">
        </script>
        <script src="../../Resources/Scripts/MadCapTopic.js" type="text/javascript">
        </script>
    </head>
    <body>
        <div class="foundation-wrap off-canvas-wrapper">
            <div class="off-canvas-wrapper-inner" data-off-canvas-wrapper="">
                <aside class="off-canvas position-left" id="offCanvas" data-off-canvas="" data-position="left" data-mc-ignore="true">
                    <ul class="off-canvas-drilldown vertical menu off-canvas-list" data-drilldown="" data-mc-back-link="Back" data-mc-css-tree-node-expanded="is-drilldown-submenu-parent" data-mc-css-tree-node-collapsed="is-drilldown-submenu-parent" data-mc-css-sub-menu="vertical menu slide-in-left is-drilldown-submenu" data-mc-include-indicator="False" data-mc-include-icon="False" data-mc-include-parent-link="True" data-mc-include-back="True" data-mc-defer-expand-event="True" data-mc-expand-event="click.zf.drilldown" data-mc-toc="True">
                    </ul>
                </aside>
                <div class="off-canvas-content inner-wrap" data-off-canvas-content="">
                    <div data-sticky-container="" class="title-bar-container">
                        <nav class="title-bar tab-bar sticky" data-sticky="" data-options="marginTop:0" style="width:100%" data-sticky-on="only screen and (max-width: 1000px)" data-mc-ignore="true">
                            <div class="middle title-bar-section outer-row clearfix">
                                <div class="menu-icon-container relative clearfix">
                                    <button class="menu-icon" data-toggle="offCanvas"><span></span>
                                    </button>
                                </div>
                            </div>
                            <div class="title-bar-layout outer-row">
                                <div class="logo-wrapper"><a class="logo" href="index.htm" alt="Logo"></a>
                                </div>
                                <div class="navigation-wrapper nocontent">
                                    <ul class="navigation clearfix" data-mc-css-tree-node-has-children="has-children" data-mc-css-sub-menu="sub-menu" data-mc-expand-event="mouseenter" data-mc-top-nav-menu="True" data-mc-max-depth="3" data-mc-include-icon="False" data-mc-include-indicator="False" data-mc-include-children="True" data-mc-include-siblings="True" data-mc-include-parent="True" data-mc-toc="True">
                                        <li class="placeholder" style="visibility:hidden"><a>placeholder</a>
                                        </li>
                                    </ul>
                                </div>
                                <div class="nav-search-wrapper">
                                    <div class="nav-search row">
                                        <form class="search" action="#">
                                            <div class="search-bar search-bar-container needs-pie">
                                                <input class="search-field needs-pie" type="search" placeholder="Search" />
                                                <div class="search-filter-wrapper">
                                                    <div class="search-filter">
                                                        <div class="search-filter-content">
                                                            <ul>
                                                                <li>All Files</li>
                                                            </ul>
                                                        </div>
                                                    </div>
                                                </div>
                                                <div class="search-submit-wrapper" dir="ltr">
                                                    <div class="search-submit" title="Search">
                                                    </div>
                                                </div>
                                            </div>
                                        </form>
                                    </div>
                                </div>
                            </div>
                        </nav>
                    </div>
                    <section class="main-section">
                        <div class="row outer-row sidenav-layout">
                            <div class="sidenav-wrapper">
                                <div class="sidenav-container">
                                    <ul class="off-canvas-accordion vertical menu sidenav" data-accordion-menu="" data-mc-css-tree-node-expanded="is-accordion-submenu-parent" data-mc-css-tree-node-collapsed="is-accordion-submenu-parent" data-mc-css-sub-menu="vertical menu accordion-menu is-accordion-submenu nested" data-mc-include-indicator="False" data-mc-include-icon="False" data-mc-include-parent-link="False" data-mc-include-back="False" data-mc-defer-expand-event="True" data-mc-expand-event="click.zf.accordionMenu" data-mc-toc="True" data-mc-side-nav-menu="True">
                                    </ul>
                                </div>
                            </div>
                            <div class="body-container" data-mc-content-body="True">
                                <h1>Getting Started</h1>
                                <p>This section of the guide provides brief examples for performing some of the most essential and valuable tasks supported by Nauta software.&#160;</p>
                                <p><b>Note</b>: Several commands and training scripts in this section require access to the internet to download data, scripts, etc.</p>
                                <p>The section discusses the following topics:</p>
                                <ul>
                                    <li value="1"><a href="#Verifyin" class="MCXref xref">Verifying Installation</a>
                                    </li>
                                    <li value="2"><a href="#Overview" class="MCXref xref">Overview of nctl Commands</a>
                                    </li>
                                    <li value="3"><a href="#Example" class="MCXref xref">Example Experiments</a>
                                    </li>
                                    <li value="4"><a href="#Adding" class="MCXref xref">Adding Experiment Metrics (Optional)</a>
                                    </li>
                                    <li value="5"><a href="#Viewing2">Viewing Experiment Results from the Web UI</a>
                                    </li>
                                    <li value="6"><a href="#Launchin">Launching Kubernetes Dashbboard</a>
                                    </li>
                                    <li value="7"><a href="#Inferenc">Inference</a>
                                    </li>
                                    <li value="8"><a href="#Launch" class="MCXref xref">Launching TensorBoard</a>
                                    </li>
                                    <li value="9"><a href="#Viewing" class="MCXref xref">Viewing Experiment Output Folder</a>
                                    </li>
                                    <li value="10"><a href="#Removing" class="MCXref xref">Removing Experiments (Optional) </a> </li>
                                </ul>
                                <h2><a name="Verifyin"></a>Verifying Installation</h2>
                                <p>Check that the required software packages are available in terminal by PATH and verified that correct version is used.</p>
                                <p><b>Note 1</b>: Docker* can contain kubectl* binary in a version that is not supported. Please verify that the PATH environment variable has the correct order of paths and that the kubectl from docker package is not used.</p>
                                <p><b>Note 2</b>: If you are behind a proxy, remember to set the  HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables. For Ubuntu, these variables must be set or the software will not work correctly.</p>
                                <p>Enter the following command to verify your installation. </p>
                                <p><span style="font-family: 'Courier New';">$ nctl verify</span>
                                </p>
                                <p>If any installation issues are found, the command returns information about their cause (which application should be installed and in which version). This command also checks if the CLI can connect to Nauta and if port forwarding to Nauta is working properly.</p>
                                <p>If no issues are found, a message indicates checks were successful.  The following is displayed:</p>
                                <p style="text-align: center;">
                                    <img src="../images/verify_results_346x195.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 346;height: 195;" />
                                </p>
                                <h2><a name="Overview"></a>Overview of nctl Commands</h2>
                                <p>Each nctl command has at least two options:</p>
                                <ul>
                                    <li value="1">-<span style="font-family: 'Courier New';">v, --verbose</span>  Set verbosity level:<ul><li value="1"><span style="font-family: 'Courier New';">-v</span> for INFO - basic logs on INFO/EXCEPTION/ERROR levels are displayed.</li><li value="2"><span style="font-family: 'Courier New';">-vv</span> for DEBUG - detailed logs on INFO/DEBUG/EXCEPTION/ERROR levels are displayed.</li></ul></li>
                                    <li value="2"><span style="font-family: 'Courier New';">-h, --help</span>     - the application displays the usage and options available for a specific command or subcommand.</li>
                                </ul>
                                <p>&#160;</p>
                                <p>Access help for any command with the <span style="font-family: 'Courier New';">--help</span> or <span style="font-family: 'Courier New';">-h</span> parameter. The following command provides a list and brief description of all nctl commands. </p>
                                <p><b>Execute</b>: <span style="font-family: 'Courier New';">nctl --help</span></p>
                                <p>The results are shown below.</p>
                                <p style="text-align: center;">
                                    <img src="../images/nctl_commands_top.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;" />
                                </p>
                                <h2><a name="Example"></a>Example Experiments</h2>
                                <p>The Nauta installation includes sample training scripts and utility scripts, contained in the <span style="font-family: 'Courier New';">examples</span> folder, that can be run to demonstrate how to use Nauta. This section describes how to use these scripts.</p>
                                <h3>Examples Folder Content</h3>
                                <p>The examples folder in the nctl installation contains these following experiment scripts:</p>
                                <ul>
                                    <li value="1"><span style="font-family: 'Courier New';">mnist_single_node.py</span> - training of digit classifier in single node setting</li>
                                    <li value="2"><span style="font-family: 'Courier New';">mnist_multinode.py</span> - training of digit classifier in distributed TensorFlow setting</li>
                                    <li value="3"><span style="font-family: 'Courier New';">mnist_horovod.py</span> - training of digit classifier in Horovod</li>
                                </ul>
                                <p>&#160;</p>
                                <p>There are also the following two utility scripts, <span style="font-family: 'Courier New';">mnist_converter_pb.p</span>y and  <span style="font-family: 'Courier New';">mnist_checker.py</span> which are related to inference process and model verification.</p>
                                <p><b>Note</b>: Experiment scripts must be written in Python.</p>
                                <h3>Launching Training Using the Scripts</h3>
                                <p>Launching training in Nauta in these examples is performed with the following command:</p>
                                <p><b>Syntax</b>:&#160;<span style="font-family: 'Courier New';">nctl experiment submit -t template SCRIPT_LOCATION</span></p>
                                <p>with:</p>
                                <ul>
                                    <li value="1"><span style="font-family: 'Courier New';">template = tf-training-tfjob and SCRIPT_LOCATION = examples/mnist_single_node.py</span> (relative to nctl root) for single node training. The template parameter in this case is optional.</li>
                                    <li value="2">or <span style="font-family: 'Courier New';">template = multinode-tf-training-tfjob and SCRIPT_LOCATION = examples/mnist_multinode.py for multinode training</span></li>
                                    <li value="3">or <span style="font-family: 'Courier New';">template = multinode-tf-training-horovod</span> and <span style="font-family: 'Courier New';">SCRIPT_LOCATION = examples/mnist_horovod.py for Horovod training</span></li>
                                </ul>
                                <p><b>Note</b>: All examples assumes macOS. Any line-wraps below are not intended.</p>
                                <h3><a name="Submitti"></a>Submitting an Experiment</h3>
                                <p><b>Note 1</b>: The included example scripts do not require external data source. They download the MNIST dataset and save it locally. Templates referenced here have some set CPU and memory requirements.</p>
                                <p><b>Note 2</b>:  Please refer to for more info about changing those if you want to.</p>
                                <p><b>Note 3</b>: For more info about experiment submit command please refer to <a href="experiment.htm#submit" class="MCXref xref">submit Subcommand</a>.</p>
                                <p>This example will show how to submit a mnist experiment and write the TensorBoard data to a folder in your Nauta output folder. Enter the following command.</p>
                                <p>Following is the basic syntax for experiment submit command:</p>
                                <p><b>Syntax</b>: <span style="font-family: 'Courier New';">nctl experiment submit [options] SCRIPT_LOCATION -- [script_parameters]</span></p>
                                <p>Enter the following command to run this example:</p>
                                <p><span style="font-family: 'Courier New';">$ nctl experiment submit -t tf-training-tfjob examples/mnist_single_node.py --name single</span>
                                </p>
                                <p>Where:</p>
                                <p>•	<span style="font-family: 'Courier New';">-t,  TEXT</span>: Name of a template that will be used by Nauta to create a description of a job to be submitted. By default, this is a single-node tensorflow training. The template is chosen. List of available templates might be obtained by Issuing <span style="font-family: 'Courier New';">nctl experiment template_list</span> command.</p>
                                <p>•	<span style="font-family: 'Courier New';">-- SCRIPT_LOCATION</span>: The path and name of the Python script use to perform this experiment.</p>
                                <h3>Result of this Command</h3>
                                <p>The execution of the submit command may take a few minutes the first time. During this time, resources are allocated, the experiment package is created using parameters, scripts, and settings, a docker image is built then pushed to the Kubernetes cluster. When the experiment submission is complete, the following result is displayed:</p>
                                <p style="text-align: center;">
                                    <img src="../images/single_exp_tensorboard_594x130.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 594;height: 130;" />
                                </p>
                                <h3>Viewing Experiment Status</h3>
                                <p>Use the following command to view the status of all your experiments:</p>
                                <p><b>Syntax</b>: <span style="font-family: 'Courier New';">nctl experiment list [options]</span></p>
                                <h4 style="font-size: 11pt;font-weight: normal;">Enter this command:</h4>
                                <p><span style="font-family: 'Courier New';">$ nctl experiment list --brief</span>
                                </p>
                                <p>Experiment status displays as below. The <span style="font-family: 'Courier New';">--brief</span> option returns a short version of results shown below. <i>This is an example only.</i></p>
                                <p style="text-align: center;">
                                    <img src="../images/experiment_status_591x210.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 591;height: 210;" />
                                </p>
                                <h2>Monitoring Training</h2>
                                <p>There are three ways to monitor training in Nauta, all which are discussed in the following sections.</p>
                                <ul>
                                    <li value="1">Viewing Experiment Logs</li>
                                    <li value="2">Adding Experiment Metrics</li>
                                    <li value="3">Viewing Experiment Results from the Web UI</li>
                                    <li value="4">Using Tensorboard</li>
                                </ul>
                                <h3>&#160;</h3>
                                <h3><a name="View2"></a>Viewing Experiment Logs</h3>
                                <p>Use the following command to view the experiment log.</p>
                                <p><b>Syntax</b>:<span style="font-family: 'Courier New';">nctl experiment logs [options] EXPERIMENT_NAME</span></p>
                                <p>Enter this command:</p>
                                <p><span style="font-family: 'Courier New';">$ nctl experiment logs mnist—tb</span>
                                </p>
                                <p>A log displays as follows. The result below is an example only.</p>
                                <p style="text-align: center;">
                                    <img src="../images/experiment_log_618x174.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 618;height: 174;" />
                                </p>
                                <h2><a name="Adding"></a>Adding Experiment Metrics (Optional)</h2>
                                <p>Experiments launched in Nauta can output additional kinds of metrics using the  publish function from the experiment metrics API. To see an example of metrics published with the single experiment executed in the above example, enter this command:</p>
                                <p><span style="font-family: 'Courier New';">$ nctl experiment list</span>
                                </p>
                                <p>Following are example results (only a fragment is shown):</p>
                                <p style="text-align: center;">
                                    <img src="../images/adding_metrics_641x170.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 641;height: 170;" />
                                </p>
                                <p>To add metrics <i>to an experiment file you have created</i>, you need to edit the experiment file to use the <span style="font-family: 'Courier New';">experiment_metrics.api</span>  and then publish the accuracy in your experiment file. Perform the following steps:</p>
                                <ol>
                                    <li value="1">Add the metrics library API with the following entry: <br /><span style="font-family: 'Courier New';">from experiment_metrics.api import publish</span><br /></li>
                                    <li value="2">Add metrics for publishing of last step's accuracy by adding this code in the ‘def feed_dict’ definition after the for loops: <br /><br /><span style="font-family: 'Courier New';">metrics = {} </span><br style="font-family: 'Courier New';" /><span style="font-family: 'Courier New';">metrics["accuracy_step_{}".format(i)] = str(acc)</span><br style="font-family: 'Courier New';" /><span style="font-family: 'Courier New';">publish(metrics)</span><br /></li>
                                    <li value="3">Save the changes.<br /></li>
                                    <li value="4">Submit the experiment again, but with a different name. <br /></li>
                                    <li value="5">
                                        <p> The published metrics can now be viewed: <br /><br /><span style="font-family: 'Courier New';">$ nctl &lt;new_name&gt; experiment list</span><br /></p>
                                    </li>
                                </ol>
                                <h3>Information About Saving Metrics for Multinode Experiments</h3>
                                <p>Storing at the same time two (or more) metrics with the same key from two different nodes may lead to errors (such as loosing some logs) due to conflicting names. To avoid this, adding metrics for multinode experiments should be done using one of the two following methods:</p>
                                <p>Method 1:&#160;The key of a certain metric should also contain a node identificator from which this metric comes. Creation of such identificator can be done in the following ways:<ul><li value="1">For horovod multinode training jobs, result of the <span style="font-family: 'Courier New';">rank()</span> function provided by the <span style="font-family: 'Courier New';">horovod</span> package can be used as a node's identificator.</li></ul><ul><li value="1">For <span style="font-family: 'Courier New';">tfjob</span> multinode training jobs, a user can take all necessary info from the TF_CONFIG environment variable. Here is an example piece of a code creating such identificator:</li></ul></p>
                                <ol>
                                    <br /><span style="font-family: 'Courier New';">tf_config = os.environ.get('TF_CONFIG')
<br />if not tf_config:<br />   raise RuntimeError('TF_CONFIG not set!')
<br />tf_config_json = json.loads(tf_config)
<br />job_name = tf_config_json.get('task', {}).get('type')<br />task_index = tf_config_json.get('task', {}).get('index')<br /># final node identificator
node_id = '-'.join(job_name,task_index)</span><![CDATA[
]]><br /></ol>
                                <p>&#160;</p>
                                <p>Method 2&#160;:Only one node should store metrics. Deciding which node should store metrics can be done in the following ways:<ul><li value="1">For <span style="font-family: 'Courier New';">horovod</span> multinode training jobs, the horovod python library provides the <span style="font-family: 'Courier New';">rank()</span> function that returns a number of a current worker. Master is marked with the number 0, so only a pod with this number should store logs.</li></ul><ul><li value="1">For <span style="font-family: 'Courier New';">tfjob</span> multinode training jobs, because there is no dedicated master node, a user should choose which worker should be responsible for storing metrics. The identificator of a current worker can be obtained as described in method 1 above. A user should choose one of such identificators and store logs only from a node having this chosen id.</li></ul></p>
                                <h2><a name="Viewing2"></a>Viewing Experiment Results from the Web UI</h2>
                                <p>The web UI lets you explore the experiments you have submitted. To view your experiments at the web UI, use the following command at the command prompt:</p>
                                <p><span style="font-family: 'Courier New';">$ nctl launch webui</span>
                                </p>
                                <p>The following screen displays  (this is an example only).</p>
                                <p>
                                    <img src="../images/web_ui.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;" />
                                </p>
                                <p>The web UI shows the six columns listed below.  Each of these column headings are clickable, so to re-sort the listing of experiments based on that column heading, ascending or descending order, click on that column heading.   Here are the six columns:</p>
                                <ul>
                                    <li value="1"><b>Name</b>: The left-most column lists the experiments by name.</li>
                                    <li value="2"><b>Status</b>: This column reveals the experiment’s current status, one of: QUEUED, RUNNING, COMPLETE, CANCELED, FAILED, CREATING.</li>
                                    <li value="3"><b>Submission Date</b>: The column gives the submission date in the format: MM/DD/YYYY, hour:min:second AM/PM.</li>
                                    <li value="4"><b>Start Date</b>: This column shows the experiment start date in the format: MM/DD/YYYY, hour:min:second AM/PM.  The Start Date (or time) will always be after the Submission Date (or time)..</li>
                                    <li value="5"><b>Duration</b>: This column shows the duration of execution for this experiment in days, hours, minutes and seconds.</li>
                                    <li value="6"><b>Type</b>: Experiment Type can be Training, Jupyter, or Inference. Training indicates that the experiment was launched from the CLI. Jupyter indicates that the experiment was launched using Jupyter Notebook. Inference means that training is largely complete and you have begun running predictions (Inference) with this model. (If the model used for inference was already present, there was no training performed on Nauta.)</li>
                                </ul>
                                <p>You can perform the tasks discussed below at the Nauta web UI.</p>
                                <h3>Expand Experiment Details</h3>
                                <p>Click on a <i>listed experiment name</i> to see additional details for that experiment.  The following details are examples only.</p>
                                <p>This screen is divided into two frames. The left-side frame shows:</p>
                                <ul>
                                    <li value="1"><b>Resources</b> assigned to that experiment, specifically the assigned pods and their status and container information including the CPU and memory resources assigned.</li>
                                    <li value="2"><b>Submission Date</b> and time.</li>
                                </ul>
                                <p>&#160;</p>
                                <p style="text-align: center;">
                                    <img src="../images/UI_Experiment_Details_1.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;" />
                                </p>
                                <p>The right-side frame of the experiment details windows shows:</p>
                                <ul>
                                    <li value="1"><b>Start Date</b>: The day and time this experiment was launched.</li>
                                    <li value="2"><b>End date</b>: The day and time this experiment was launched.</li>
                                    <li value="3"><b>Total Duration</b>: The actual duration this experiment was instantiated.</li>
                                    <li value="4"><b>Parameters</b>: The experiment script file name and the log directory.</li>
                                    <li value="5"><b>Output</b>: Clickable links to download all logs and view the last 100 log entries.</li>
                                </ul>
                                <p>&#160;</p>
                                <p style="text-align: center;">
                                    <img src="../images/UI_Experiment_Details_2.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;" />
                                </p>
                                <h3><a name="Searchin"></a>Searching on Experiments</h3>
                                <p>In the <b>Search</b> field <img src="../images/search_lense.png" style="width: 32px;height: 32px;" />at the far right of the UI, enter a string of alphanumeric characters to match the experiment name or other parameters (such as user), and list only those matching experiments.  This Search function lets the user search fields in the entire list, not just the experiment name or parameters.</p>
                                <h3><a name="Add/Dele"></a>Adding and Deleting Columns</h3>
                                <p>Click <b>ADD</b>/<b>DELETE COLUMNS</b> to open a scrollable dialogue. Here, the columns currently in use are listed first with their check box checked. Scroll down to see more, available columns listed next, unchecked. Click to check and uncheck and select the column headings you prefer. Optional column headings include parameters such as Pods, End Date, Owner, Template, Time in Queue, etc.</p>
                                <p>Column headings also include metrics that have been setup using the Metrics API, for a given experiment, and you can select to show those metrics in this display as well.</p>
                                <p>Refer to <a href="view_exp_tensorbd.htm">Launching TensorBoard to View Experiments</a>.</p>
                                <h2><a name="Launchin"></a>Launching Kubernetes</h2>
                                <p>Click the “hamburger menu” <img src="../images/hamburger_menu.png" /> at the far left of the UI to open a left frame. Click <b>Resources Dashboard</b> to open the Kubernetes resources dashboard. Refer to <a href="accessing_kubernetes.htm">Accessing the Kubernetes Resources Dashboard</a>. </p>
                                <h2><a name="Launch"></a>Launching TensorBoard</h2>
                                <p>Generally, every file that the training script outputs to /mnt/output/experiment (accessed from the perspective of training script launched in Nauta) is accessible from the outside after mounting the output directory with command provided by nctl mount.  (Refer to  Mounting Storage to View Experiment Output for more information.)</p>
                                <p>When training scripts output Tensorflow summaries to /mnt/output/experiment, they can be automatically picked up by TensorBoard instance launched with command:</p>
                                <h4>Syntax:</h4>
                                <p><span style="font-family: 'Courier New';">nctl launch tensorboard [options] EXPERIMENT_NAME</span>
                                </p>
                                <p>Enter this command:</p>
                                <p><span style="font-family: 'Courier New';">$ nctl launch tensorboard single</span>
                                </p>
                                <p>The following message displays. The port number is an example only.</p>
                                <p style="text-align: center;">
                                    <img src="../images/tensorboard_from_cli_417x83.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 417;height: 83;" />
                                </p>
                                <p>Figure 1 shows the browser display of TensorBoard dashboard with the experiment’s results.</p>
                                <p style="text-align: center;"><b>Figure 1: TensorBoard Dashboard - Example Only</b>
                                </p>
                                <p style="text-align: center;">
                                    <img src="../images/tensorboard.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;" />
                                </p>
                                <h2><a name="Inferenc"></a>Inference</h2>
                                <p>To perform inference testing (using predict batch command in this example) you need to:</p>
                                <ol>
                                    <li value="1">Prepare data for model input.</li>
                                    <li value="2">Acquire a trained model.</li>
                                    <li value="3">Run prediction instance with trained model on this data.</li>
                                </ol>
                                <h2>Data Preparation</h2>
                                <p>The example <span style="font-family: 'Courier New';">mnist_converter_pb.py</span> script located in the examples folder can be used for data preparation. This script prepares the sample of MNIST test set and converts it to <span style="font-family: 'Courier New';">protobuf</span> requests acceptable by the served model. This script is run locally and requires <span style="font-family: 'Courier New';">tensorflow</span>, <span style="font-family: 'Courier New';">numpy</span>, and <span style="font-family: 'Courier New';">tensorflow_serving</span> modules. The <span style="font-family: 'Courier New';">mnist_converter_pb.py</span> script takes two input parameters:</p>
                                <ul>
                                    <li value="1"><span style="font-family: 'Courier New';">--work_dir</span> which defaults to <span style="font-family: 'Courier New';">/tmp/mnist_test</span>. It is a path to directory used as <span style="font-family: 'Courier New';">workdir</span> by this script and <span style="font-family: 'Courier New';">mnist_checker.py</span>. Downloaded MNIST dataset will be stored there as well as converted test set sample and labels cached for them.</li>
                                    <li value="2"><span style="font-family: 'Courier New';">--num_tests</span> which defaults to 100. It is a number of examples from test set which will be converted. Max value is 10000</li>
                                </ul>
                                <p>Running:</p>
                                <p><span style="font-family: 'Courier New';">python examples/mnist_converter_pb.py</span>
                                </p>
                                <p>will create <span style="font-family: 'Courier New';">/tmp/mnist_test/conversion_out</span> folder, fill it with 100 <span style="font-family: 'Courier New';">protobuf</span> requests, and cache labels for these requests in <span style="font-family: 'Courier New';">/tmp/mnist_test/labels.npy</span> file.</p>
                                <h3>Trained Model</h3>
                                <p>Servable models (as with other training artifacts) can be saved by a training script. As previously mentioned, to access these you have to use the command provided by the <span style="font-family: 'Courier New';">nctl mount</span> command and mount output storage locally. Example scripts all save servable models in their models subdirectory. To use models like this for inference, you will have to mount input storage too, because models have to be accessible from inside of the cluster.</p>
                                <p>For the single experiment example, execute these commands:</p>
                                <blockquote>
                                    <blockquote>
                                        <p style="font-family: 'Courier New';">$ mkdir /mnt/input<br />$ mkdir /mnt/input/single<br />$ mkdir /mnt/output<br />$ ... mount command provided with nctl mount used to mount output storage to /mnt/output<br />$ ... mount command provided with nctl mount used to mount input storage to /mnt/input<br />$ cp /mnt/output/single/models/* -Rf /mnt/input/single/</p>
                                    </blockquote>
                                    <p>After these steps <span style="font-family: 'Courier New';">/mnt/input/single</span> should contain:</p>
                                    <blockquote>
                                        <p style="font-family: 'Courier New';">/mnt/input/single/:<br />00001<br /><br />/mnt/input/single/00001:<br />saved_model.pb  variables<br /><br />/mnt/input/single/00001/variables:<br />variables.data-00000-of-00001  variables.index</p>
                                    </blockquote>
                                </blockquote>
                                <h3>Running Prediction Instance</h3>
                                <p>The following provides a brief example of running inference using the <span style="font-family: 'Courier New';">batch</span> command. For more information, refer to <a href="inference_testing.htm" class="MCXref xref">Evaluating Experiments with Inference Testing</a>.</p>
                                <p>Before running the <span style="font-family: 'Courier New';">batch</span> command, you need to copy <span style="font-family: 'Courier New';">protobuf</span> requests to input storage, because  they need to be accessed by the prediction instance too.</p>
                                <p>Enter these commands:</p>
                                <blockquote>
                                    <p style="font-family: 'Courier New';">$ mkdir /mnt/input/data<br />$ cp /tmp/mnist_test/conversion_out/* /mnt/input/data</p>
                                </blockquote>
                                <p>The next command will create a prediction instance.:</p>
                                <p><span style="font-family: 'Courier New';">$ nctl predict batch -m /mnt/input/home/single -d /mnt/input/home/data --model-name mnist --name single-predict</span>
                                </p>
                                <p>Following are the example results of this command:</p>
                                <p style="text-align: center;">
                                    <img src="../images/predict_instance_566x93.png" style="border-left-style: solid;border-left-width: 1px;border-left-color: ;border-right-style: solid;border-right-width: 1px;border-right-color: ;border-top-style: solid;border-top-width: 1px;border-top-color: ;border-bottom-style: solid;border-bottom-width: 1px;border-bottom-color: ;width: 566;height: 93;" />
                                </p>
                                <p><b>Note 1</b>: Notice the additional home directory in path to both model and input data. This is how the path looks from the perspective of the prediction instance.</p>
                                <p><b>Note 2</b>: <span style="font-family: 'Courier New';">mnist_converter_pb.py</span> creates requests to the mnist model. Note that <span style="font-family: 'Courier New';">--model-name mnist</span> is where this mnist name is given to the prediction instance.</p>
                                <p><b>Note 3</b>: For more info about predict command, please refer to <a href="predict.htm" class="MCXref xref">predict Command</a>.</p>
                                <p>After the prediction instance completes (can be checked using the <span style="font-family: 'Courier New';">predict list</span> command), you can collect instance responses from <span style="font-family: 'Courier New';">output</span> storage. In our example, it would contain 100 <span style="font-family: 'Courier New';">protobuf</span> responses. These can be validated in our case using <span style="font-family: 'Courier New';">mnist_checker.py</span>. </p>
                                <p>Running the following command locally will display the error rate calculated for this model and this sample of the test set:</p>
                                <p><span style="font-family: 'Courier New';">python examples/mnist_checker.py --input_dir /mnt/output/single-predict</span>
                                </p>
                                <h2><a name="Viewing"></a>Viewing Experiment Output Folder</h2>
                                <p>You can use the following steps to mount the output folder and view TensorBoard data files.</p>
                                <p>Mount a folder to your NAUTA namespace output directory:</p>
                                <ol>
                                    <li value="1">macOS/Ubuntu: First, mount your NAUTA output directory to a local folder. Create a folder for mounting named my_output_folder:<br /><span style="font-family: 'Courier New';">$ mkdir my_output_folder</span></li>
                                    <li value="2">To see the correct mount options and command:<br /><span style="font-family: 'Courier New';">$ nctl mount</span></li>
                                    <li value="3">Use the mounting command that was displayed to mount Nauta storage to  your local machine. Following are examples of mounting the local folder to the Nauta output folder for each OS:<ul><li value="1">MacOS:&#160;<span style="font-family: 'Courier New';">mount_mbfs //'USERNAME:PASSWORD'@CLUSTER-URL/output my_output_folder</span></li><li value="2"><p>Ubuntu:&#160;<span style="font-family: 'Courier New';">sudo mount.cifs  -o username=USERNAME,password=PASSWORD,rw,uid=1000 //CLUSTER-URL/output my_output_folder</span></p></li><li value="3"><p>Windows:&#160; Use Y: drive as mount point. <span style="font-family: 'Courier New';">net case Y: \\CLUSTER-URL\output /user:USERNAME PASSWORD</span></p></li></ul></li>
                                    <li value="4">
                                        <p>Navigate to the mounted location</p>
                                        <ul>
                                            <li value="1">MacOS/Ubuntu only: Navigate to my_output_folder.</li>
                                            <li value="2">Windows only:&#160;Open Explorer Window and navigate to Y: drive.</li>
                                        </ul>
                                    </li>
                                    <li value="5">
                                        <p>See the saved event file by navigating to <span style="font-family: 'Courier New';">mnist-single-node/tensorboard</span>. Example file: <span style="font-family: 'Courier New';">events.out.tfevents.1542752172.mnist-single-node-master-0</span></p>
                                    </li>
                                    <li value="6">Unmount  Nauta storage using one of the below commands:<ul><li value="1">MacOS: <span style="font-family: 'Courier New';">umount output my_output_folder</span></li><li value="2">Ubuntu: s<span style="font-family: 'Courier New';">udo umount my_output_folder</span></li><li value="3">Windows: Eject or <span style="font-family: 'Courier New';">net use Y: /delete</span><br /><br /></li></ul></li>
                                </ol>
                                <p>To unmount previously mounted Nauta input storage from a local folder/machine refer to <a href="unmount.htm">Unmounting Experiment Input to Storage.</a></p>
                                <p>For more information on mounting, refer to <a href="working_with_datasets.htm" class="MCXref xref">Working with Datasets</a>.</p>
                                <h2><a name="Removing"></a>Removing Experiments (Optional) </h2>
                                <p>An experiment that has been completed and is no longer needed can be removed from the experiment list using the cancel command and its purge option. The experiment will only be removed from the experiment list. The experiment’s artifacts will remain in the Nauta storage output folder. Logs will be removed.</p>
                                <p><b>Syntax</b>:</p>
                                <p><span style="font-family: 'Courier New';">nctl experiment cancel [options] EXPERIMENT_NAME</span>
                                </p>
                                <p style="font-weight: normal;"> <span style="font-family: 'Courier New';">$ nctl experiment cancel –-purge &lt;your_experiment&gt;</span></p>
                                <p>&#160;</p>
                                <p>&#160;</p>
                                <p>&#160;</p>
                                <p>&#160;</p>
                            </div>
                        </div>
                    </section><a data-close="true"></a>
                </div>
            </div>
            <script>/* <![CDATA[ */$(document).foundation();/* ]]> */</script>
        </div>
    </body>
</html>